Spin-liquid versus spiral-order phases in the anisotropic triangular lattice 
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We study the competition between magnetic and spin-liquid phases in the Hubbard model on the 
anisotropic triangular lattice, which is described by two hopping parameters t and t' in different 
spatial directions and is relevant for layered organic charge-transfer salts. By using a variational 
approach that includes spiral magnetic order, we provide solid evidence that a spin-liquid phase 
is stabilized in the strongly-correlated regime and close to the isotropic limit t' jt — 1. Otherwise, 
a magnetically ordered spiral state is found, connecting the (collinear) Neel and the (coplanar) 
120° phases. The pitch vector of the spiral phase obtained from the unrestricted Hartree-Fock 
approximation is substantially renormalized in presence of electronic correlations, and the Neel 
phase is stabilized in a wide regime of the phase diagram, i.e., for t' /t < 0.75. We discuss these 
results in the context of organic charge-transfer salts. 

PACS numbers: 71.10.Fd, 71.27.+a, 75.10.-b 
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Introduction- The combined presence of strong elec- 
tron interaction and geometrical frustration leads to a 
plethora of interesting phenomena, like superconductiv- 
ity, metal-insulator (Mott) transition, or purely quantum 
paramagnets, the so-called spin liquids. In this context, 
the organic charge-transfer salts k-(ET)2X [l| play an 
important role. 0, S| A large variety of phases have been 
found when changing temperature, pressure or the nature 
of the anion X, ranging from correlated metals with su- 
perconductivity at low temperatures, to insulators with 
magnetic order. 0-0| Even more interestingly, a metal- 
insulator transition to a pure non-magnetic Mott insu- 
lating state has been detected for the compound with 
X=Cu 2 (CN) 3 . Recently, another family of organic 

materials, denoted by Et„Me4_„Pn[Pd(dmit)2]2, [13] 
has been shown to display different quantum phases, in- 
cluding valence-bond solid and spin- liquid states, fll, 12] 
From a quantum chemical perspective, the simplest pos- 
sible effective Hamiltonian for the organic charge-transfer 
salts is the Hubbard model (after an appropriate particle- 
hole transformation) on the anisotropic triangular lat- 
tice at half filling. Indeed, in these materials, strongly 
dimerized organic molecules are arranged in stacked two- 
dimensional triangular lattices; each dimer has (on the 
average) a charge state with one hole, implying a half- 
filled conducting band. In addition, a sizable effective 
Coulomb repulsion is felt by two holes on the same dimer, 
while longer range correlations are much smaller. 13l.ll4| 

The Hubbard model is defined by: 



U y2 n i,t n i,h (!) 



where c\ a {ci >a ) creates (destroys) an electron with spin 




a on site i, rii^ = c\ a Ci.c, is the electronic density, Uj 
is the hopping amplitude and U is the on-site Coulomb 



FIG. 1: Illustration of the anisotropic triangular lattice in the 
square topology (left), used in this work, and in the equivalent 
triangular topology (right). Solid and dashed lines denote 
hopping amplitudes t and t' , respectively. 



repulsion. In this work, we focus our attention on the 
half-filled case, where the number of electrons N e equals 
the number of sites L, and consider a square lattice with 
a nearest-neighbor hopping t, along (1,0) and (0,1) di- 
rections, and a further next-nearest-neighbor hopping t' 
along (1,1); this choice of the hopping amplitudes is topo- 
logically equivalent to the anisotropic triangular lattice, 
see Fig. Q] According to recent density functional theory 
calculations, 13- li| the ratio t' jt appropriate for organic 
salts lies in the range [0.3, 1.3], with the spin-liquid com- 
pound k-(ET) 2 Cu 2 (CN) 3 located at t' /t ~ 0.83. 

A major issue in the Hubbard model on the anisotropic 
triangular lattice is the possibility of stabilizing a spin- 
liquid phase, compatible with the experimental data. 
Several approaches have proposed the existence of a spin 
liquid-region for t' jt < 1, based on path-integral renor- 
malization group (PIRG), [13] dynamical mean- field the- 
ory (DMFT), [3] exact diagonalization, [l!| and varia- 
tional Monte Carlo (VMC). [20J In addition, an exact di- 
agonalization study based upon a strong-coupling expan- 
sion suggested a possible spin-liquid phase even for the 
isotropic case t'/t = 1, close to the metal-insulator tran- 
sition. [2l| However, for generic values of the ratio t'/t, 
magnetic states with incommensurate order may be ex- 
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pected and indeed have been proposed by different mean- 
field approaches, iike for instance within the Hartree- 
Fock (HF) approximation [22 - 24 or the renormalized 
mean-field method. [25[ However, due to the difficulty of 
constructing such magnetic states with generic ordering 
vectors, none of these studies was able to perform a fair 
comparison between the spin-liquid phase and the spiral 
states that have been proposed. In this respect, recent 
density-matrix renormalization group (DMRG) calcula- 
tions, implemented with suitable boundary conditions, 
have been able to deal with incommensurate magnetism 
in the Heisenberg model. (26| 

In order to go beyond the previous studies, we ap- 
proach this problem by implementing correlated vari- 
ational wave functions which describe magnetic states 
with generic incommensurate order. This can be achieved 
by starting from the spiral states obtained at the HF 
level and including, in a second step, many body corre- 
lations. In this way, we are able to treat incommensu- 
rate spiral order and non-magnetic states on the same 
level and determine which state is stabilized for a given 
value of frustration t'/t and Coulomb repulsion U. Vari- 
ational approaches may contain, as a matter of principle, 
a bias towards ordered states. However, we showed pre- 
viously 0, [53] that very accurate results are obtained 
in a wide regime of the parameters when using, as in the 
present work, generalized Gutzwiller wave functions with 
long-range Jastrow correlations and backflow corrections. 

Here, we confirm that a spin-liquid phase is favored for 
t'/t ~ 0.85, while magnetic spiral order becomes compet- 
itive close to the isotropic point, i.e., for t'/t ~ 0.95, and 
for t'/t ~ 0.75 — 0.8. In addition, we explicitly study 
the effect of correlations on a mean-field state showing 
how the energy and the pitch vector of the spiral state 
are modified when electronic correlations are taken into 
account by the VMC method. 

Unrestricted Hartree Fock- The unrestricted HF state 
is obtained by performing a mean-field decoupling of 
Eq. ©: 



#HF = - ^ C U C J> + hx - 

i 
i 



(2) 



-U 



which contains 4L independent mean-field parameters to 
be computed self-consistently: (ni.f), {^,1), ( c l^ c i,l)i 

and (c\,Cirf) for each site. Here, we slightly restrict 
the variational freedom and impose the spin order to 
be coplanar in the x—y plane, namely, we look for so- 
lutions with (ni^) — (n^i), thus reducing the number of 
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FIG. 2: (Color online) HF (left) and VMC (right) energies 
of the spiral state, for U/t = 16, as a function of the pitch 
angle 9 (in unit of 7r), for t'/t — 0.85 (top), 0.9 (middle) and 
0.95 (bottom). Cluster of sizes I x I have been used, with 
I = 12, 14, 16, 18, 20. 



independent parameters to 3L. The mean- field Hamil- 
tonian ^ can be diagonalized and the ground state 
|SP) can be computed by filling the lowest-energy single- 
particle orbitals. 

Here, we are interested in describing the nature of 
the insulating state, which is stabilized for sufficiently 
large on-site interactions. In this regime, the optimal 
HF solutions display a spiral magnetic order, which, for 
t'/t < 1, may be parametrized through a single pitch an- 
gle 9 G [2vt/3,/t]. Indeed, nearest-neighbor spins, along 
(1,0) and (0, 1) directions, form an angle 9, while next- 
nearest-neighbor spins, along the (1,1) direction, form 
an angle 29; a pitch angle of 9 = tt corresponds to Neel 
order, suitable for t' — 0, and 9 = 2ir/3 to the 120° order, 
suitable for t' = t. On finite-size clusters with periodic- 
boundary conditions, only the set of commensurate pitch 
angles is accessible; for L = I x I, the allowed values are 
9 = 2irn/l, with n being an integer. 

Variational Monte Carlo- Within the VMC approach, 
we construct magnetic states with spiral order by ap- 
plying correlation terms on top of spiral states. Since 
the optimal pitch angle in the presence of electron cor- 
relations may differ from the one obtained at the HF 
level, several different values of 9 are considered in 
the VMC calculations. We employ a spin-spin Jas- 
trow factor to correctly describe fluctuations orthogo- 
nal to the plane where the magnetic order lies, i.e., 



J s = exp[l/2£ 



QZ QZ 

hJ J i °j . 



A further density- 

: Vi^jTZirij 



density Jastrow factor J c = exp[l/2^ i3 VijUiUj] (that 
includes the on-site Gutzwiller term v^i) is considered to 
adjust electron correlations. All the UijS and the Wjj's 
are optimized for every independent distance \i — j\. The 
correlated state is then given by | \&sp ) = J7si7c|SP). 
In order to describe a non-magnetic insulator we con- 
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FIG. 3: (Color online) Upper panel: HF (red circles) and 
VMC (blue squares) energies of the optimal spiral state as a 
function of t'/t for U/t = 16. Lower panel: the pitch angle 
6 (in unit of -n) of the optimal spiral state as a function of 
t'/t for U/t — 16. The dotted horizontal line corresponds 
to 8 — 2n/3, suitable for the isotropic point t'/t = 1. The 
error bars in the VMC calculations are due to the finite-size 
limitations in the accessible pitch angles. 



struct, in a first step, an uncorrelated wave function given 
by the ground state |BCS) of a superconducting BCS 
Hamiltonian: 129H33 



HbCS = & C k,a C k,<r + AkC tt C 



k.a 



(3) 



where both the free-band dispersion and the pairing 
amplitudes are variational functions. We use the 
parametrization 

£fc = — 2?(cos k x + cos k y ) — 2? cos(k x + k y ) — /i, (4) 
A fe = 2 Abcs (cos fca, - cosky), (5) 

where the effective hopping amplitude t' , the effective 
chemical potential fi, and the pairing field Abcs are 
variational parameters to be optimized. The correlated 
state I^bcs) = Jc|BCS) allows then to describe a non- 
magnetic Mott insulator for a sufficiently singular Jas- 
trow factor v q ~ l/q 2 (v q being the Fourier transform of 
v itj ). ® 

A size-consistent and efficient way to further improve 
the correlated states I^bcs) and I^sp) is based on back- 
flow correlations. In this approach, each orbital that 
defines the unprojected states |BCS) and |SP) is taken 
to depend upon the many-body configuration, in order 
to incorporate virtual hopping processes. [27j All results 
presented here are obtained by fully incorporating the 
backflow corrections and optimizing individually every 
variational parameter in and A&, in the Jastrow fac- 
tors J c and J s , as well as in the backflow corrections. 
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FIG. 4: (Color online) Upper panel: VMC energy of the op- 
timal spiral (blue squares) and spin-liquid (black diamonds) 
states as a function of the inverse system size 1/L, with L 
ranging from 10 x 10 to 20 x 20. Data refer to the case 
t'/t = 0.85 and U/t = 16. Lower panel: the same as in 
the upper panel but for t'/t — 0.95. 



Finally, we want to stress that the best symmetry of 
the pairing A& is d x 2_ y 2, see Eq. ([5]), for all values of 
t'/t < 1. This finding is in agreement with previous 
VMC studies 34 , 3j| carried out using variational wave 
functions not containing long-range Jastrow and backflow 
correlations. 

Evolution of the pitch angle- In Fig. [21 the energy per 
site of the spiral state is presented as a function of the 
pitch angle 6, for different values of the frustrating hop- 
pings t'/t and U/t = 16, both for the HF and the VMC 
calculations. We mention that the pitch angle is only 
weakly dependent on U/t in the insulating region. For fi- 
nite lattices the set of allowed pitch angles is determined 
by commensur ability, we therefore include in Fig. [5] re- 
sults for several cluster sizes. The overall behavior of the 
energy per site versus 9 is smooth, indicating that size ef- 
fects are under control. In the region of 0.8 < t'/t < 0.9, 
a very shallow energy landscape is observed in VMC, 
while, for larger values of the frustrating hopping, the 
minimum is much more pronounced. 

We find that Jastrow and backflow terms influence the 
periodicity of the spiral order and that the inclusion of 
the correlation factors induce a sizable gain in the energy 
per site, strongly improving the quality of the variational 
state. In Fig. [3l we present the evolution of the energy 
and of the pitch angle 9 of the optimal spiral state as a 
function of t'/t, for U/t — 16. In the intermediate range 
of frustration 0.75 < t'/t < 0.9, the explicit treatment of 
electronic correlations (within the VMC level) renormal- 
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FIG. 5: (Color online) The energy (in unit of J = 4t 2 /U) 
as a function of t/U for the HF approximation (red circles), 
the optimal spiral state in VMC (blue squares), and the spin- 
liquid state (black diamonds) for t'/t = 0.8 (top), 0.9 (middle) 
and 0.95 (bottom). 



izes the angle of the HF optimal spiral state and values 
much closer to 7r are stabilized. As a result, correlation 
effects stabilize the Neel phase (i.e., 9 = ir) in a wider 
regime, e.g., for t' /t < 0.75. Close to the isotropic point, 
i.e., for t'/t < 1.0, the optimal pitch angle shifts rapidly 
towards 27r/3. 

Spiral vs. spin-liquid state- Let us now move to the 
main result of the present work and compare the op- 
timal spiral and spin-liquid states. We find that for 
t'/t = 0.85 and U/t = 16, which are suitable parame- 
ters for k-(ET)2Cu2(CN)3, the lowest variational energy 
is achieved by a magnetically disordered state, see Fig.|4] 
The results are only weakly dependent on the cluster size. 
On the other hand, for a larger value of the frustrat- 
ing hopping, i.e., t'/t — 0.95, a spiral state with angle 
9/tt ~ 0.7 is favored over the spin liquid. In this case, 
although slightly larger size effects are present for the 
spiral state, the trend is clear. 

In Fig. [5j we present the VMC energies (in unit of 
J = At 2 /U) for the optimal spiral and spin-liquid states 
as a function of U/t and different values of t'/t. We find 
that there is a critical U ft above which the spin-liquid 
state is energetically favored, while for smaller values of 




FIG. 6: (Color online) Schematic phase diagram of the Hub- 
bard model on the anisotropic triangular lattice, as obtained 
by VMC: metal (blue), insulator with magnetic Neel order 
(red), insulator with spiral magnetic order (gradient red- 
yellow-green), and spin liquid (cyan). The 120° ordered state 
with 9 — 2tt/3 (vertical green line) is stable only for t'/t = 1. 
The spiral state illustrated with the red-yellow gradient has 
a pitch angle 9 ranging from n to 0.9-7T. The border between 
the spin- liquid and the magnetic state for t'/t < 0.8 (dashed 
cyan-white line) is only inferred since pitch angles too close 
to 7r could not be resolved. 



U/t the magnetically ordered state is stabilized. The 
simple HF energy of the spiral state is also reported for 
comparison; however, the accuracy of the HF state is not 
sufficient to study the competition with the spin-liquid 
state. 

Phase diagram- In Fig. [5J we present the final phase 
diagram. We identif y a metallic phase, which is likely 
not superconductive, |2fj|.l36l] and three insulating phases: 
a phase with commensurate Neel order (i.e., 9 — 7r), a 
spiral-order phase with 27r/3 < 9 < 7r, and a spin- liquid 
region. Note that the 120° order with 9 = 2ir/3 is stable 
only at the isotropic point t'/t = 1. Within the region 
0-7 ^ t'/t 0-8 the pitch angle is close to ir, and our 
numerical calculations applied to finite-size clusters can- 
not resolve the actual value of the optimal 9. Our results 
indicate that the spin-liquid state is not stable asymp- 
totically close to the isotropic point, for t' — > t, at least 
not for any finite U/t. Finally, we would like to mention 
the fact that, within our present approach, the transition 
from a spiral state to the spin liquid is always first order. 
However, we cannot exclude the existence of a contin- 
uous transition in the exact ground state. In this re- 
gard, it would be possible to determine the nature of the 
transition by considering BCS-spiral uncorrelated states 
|BCS,SP), which are however technically extremely de- 
manding, and beyond the scope of the present study. 

Conclusions- By using a state-of-the-art variational 
approach, we studied the insulating phase of the half- 
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filled Hubbard model on anisotropic triangular lattices 
with t'/t < 1. Through a combined HF and VMC ap- 
proach, we showed that spiral states, with non-trivial 
pitch angles, are stable in the strongly frustrated re- 
gion. For larger values of interaction a spin-liquid phase 
emerges. These results open two intriguing possibilities. 
On one side, the spin-liquid and the spiral phases may 
be considered two competing phases. In this case, the 
transition would be expected to be of first order. On 
the other side, the possibility of a new route towards a 
non-magnetic correlated state emerges, namely the spin 
liquid may be considered as an instability emerging from 
a strongly correlated spiral phase. In this second 
order transition would be expected. To resolve this ques- 
tion one would need to consider strongly-correlated com- 
bined BCS-spiral states, which are however technically 
very demanding and left for future studies. Finally, in the 
parameter region of relevance for the reported spin-liquid 
behavior in organic charge-transfer salts, we also find the 
spin-liquid phase to be the most stable. Regarding spi- 
ral order, to our knowledge no experimental evidence for 
this phase has been reported for organic charge-transfer 
salts. It would be desirable to search for such a state, es- 
pecially in the context of new orderings observed in these 
materials. [37} 

L.F.T. and H.F. acknowledge the support of the Ger- 
man Science Foundation through the grant SFB/TRR49. 



[1] 
[2] 
[3] 
M 
[5] 

[6] 

[7] 

[8] 
[9] 
[10] 



ET=BEDT-TTF stands for bis(ethylenedithio)- 
tetrathiafulvalene and X is a monovalent anion. 
K. Kanoda and R. Kato, Annual Rev. of Condens. Matter 
Phys. 2, 167 (2011). 

B.J. Powell and R.H. McKenzie, Rep. Prog. Phys. 74, 
056501 (2011). 

K. Miyagawa, A. Kawamoto, Y. Nakazawa, and K. Kan- 
oda, Phys. Rev. Lett. 75, 1174 (1995). 
H. Elsinger, J. Wosnitza, S. Wanka, J. Hagel, D. 
Schweitzer, and W. Strunz, Phys. Rev. Lett. 84, 6098 
(2000). 

S. Lefebvre, P. Wzietek, S. Brown, C. Bourbonnais, D. 
Jerome, C. Meziere, M. Fourmigue, and P. Batail, Phys. 
Rev. Lett. 85, 5420 (2000). 

P. Limelette, P. Wzietek, S. Florens, A. Georges, T.A. 

Costi, C. Pasquier, D. Jerome, C. Meziere, and P. Batail, 

Phys. Rev. Lett. 91, 016401 (2003). 

Y. Shimizu, K. Miyagawa, K. Kanoda, M. Maesato, and 

G. Saito, Phys. Rev. Lett. 91, 107001 (2003). 

R.S. Manna, M. de Souza, A. Briihl, J. A. Schlueter, and 

M. Lang, Phys. Rev. Lett. 104, 016403 (2010). 

Me stands for methyl, Et stands for ethyl, Pn denotes 



[11 

[12; 

[13; 

[14 

[15; 

[is; 
[17; 
[is: 

[19 

[20 

[21 

[22 

[23 

[24 

[25 

[26 

[27 

[28 

[29 
[30 

[31 
[32 

[33 

[34 

[35 

[36 

[37 



a pnictogen atom and dmit is l,3-dithiole-2-thione-4,5- 
dithiolate. 

J.-I. Yamaura, A. Nakao, and R. Kato, J. Phys. Soc. Jpn. 
73, 976 (2004). 

Y. Shimizu, H. Akimoto, H. Tsujii, A. Tajima, and R. 

Kato, Phys. Rev. Lett. 99, 256403 (2007). 

H.C. Kandpal, I. Opahle, Y.-Z. Zhang, H.O. Jeschke, and 

R. Valenti, Phys. Rev. Lett. 103, 067004 (2009). 

K. Nakamura, Y. Yoshimoto, T. Kosugi, R. Arita, and 

M. Imada, J. Phys. Soc. Jpn. 78, 083710 (2009). 

E. P. Scriven and B.J. Powell, Phys. Rev. Lett. 109, 
097206 (2012). 

K. Nakamura, Y. Yoshimoto, and M. Imada, 
larXiv: 1208.39541 (2012). 

H. Morita, S. Watanabe, and M. Imada, J. Phys. Soc. 
Jpn. 71, 2109 (2002). 

B. Kyung and A.-M. S. Tremblay, Phys. Rev. Lett. 97, 
046402 (2006). 

R.T. Clay, H. Li, and S. Mazumdar, Phys. Rev. Lett. 
101, 166403 (2008). 

L.F. Tocchio, A. Parola, C. Gros, and F. Becca, Phys. 
Rev. B 80, 064419 (2009). 

H.-Y. Yang, A.M. Lauchli, F. Mila, and K.P. Schmidt, 
Phys. Rev. Lett. 105, 267204 (2010). 
HR. Krishnamurthy, C. Jayaprakash, S. Sarker, and W. 
Wenzel, Phys. Rev. Lett. 64, 950 (1990). 

C. Pinettes and C. Lacroix, Solid State Commun. 85, 565 
(1993). 

M. Inui and P.B. Littlewood, Phys. Rev. B 44, 4415 
(1991). 

B. J. Powell and R.H. McKenzie, Phys. Rev. Lett. 98, 
027005 (2007). 

A. Weichselbaum and SR. White, Phys. Rev. B 84, 
245130 (2011). 

L.F. Tocchio, F. Becca, and C. Gros, Phys. Rev. B 83, 
195138 (2011). 

F. Becca, M. Capone, and S. Sorella, Phys. Rev. B 62, 
12700 (2000). 

C. Gros, Phys. Rev. B 38, 931(R) (1988). 

F.C. Zhang, C. Gros, T.M. Rice, and H. Shiba, Super- 

cond. Sci. Technol. 1, 36 (1988). 

C. Gros, Annals of Physics 189, 53 (1989). 

B. Edegger, V.N. Muthukumar, and C. Gros, Adv. Phys. 
56, 927 (2007). 

M. Capello, F. Becca, M. Fabrizio, S. Sorella, and E. 
Tosatti, Phys. Rev. Lett. 94, 026406 (2005). 
J. Liu, J. Schmalian, and N. Trivedi, Phys. Rev. Lett. 
94, 127003 (2005). 

T. Watanabe, H. Yokoyama, Y. Tanaka, and J. Inoue, 
Phys. Rev. B 77, 214505 (2008). 

S. Dayal, R.T. Clay, and S. Mazumdar, Phys. Rev. B 85, 
165141 (2012). 

P. Lunkenheimer, J. Miiller, S. Krohns, F. Schrettle, A. 
Loidl, B. Hartmann, R. Rommel, M. de Souza, C. Hotta, 
JA. Schlueter, and M. Lang, Nature Materials 11, 755 
(2012). 



